TRiM: Tensor Reduction in Memory

Personalized recommendation systems are gaining significant traction due to their industrial importance. An important building block of consists what is known as the embedding layers, which exhibit a highly memory-intensive characteristics. Fundamental primitives layers vector gathers followed by reductions, low arithmetic intensity and becomes bottlenecked memory throughput. To address this issue, recent proposals in research space employ near-data processing (NDP) solution at DRAM rank-level, achieving performance speedup. We observe that prior NDP solutions based on rank-level parallelism leave left table, they do not fully reap abundant data transfer throughput inherent datapaths. Based observation datapath has hierarchical tree structure, we propose novel, fine-grained architecture for systems, augments with an “in-DRAM” reduction unit DDR4/5 rank/bank-group/bank level, improvements over state-of-the-art approaches. also hot embedding-vector replication alleviate load imbalance across units.